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Abstract 

This study examined the robustness of the estimation of fixed-effects in 
multilevel analysis, as might occur when conducting school-effects studies with 
outlying schools. Outlying values for both intercepts and slopes for individual 
schools were modeled separately to determine the effects of extreme values of 
second-level variables on the fixed-effect parameter estimation. Under the 
conditions investigated in this study, adding a cluster of outlying schools had little 
effect on the estimation of yio and yn, however the standard error of yoi increased, 
thereby increasing the conservativeness of the test of significance of yoi. This 
occurred for clusters of outlying slopes, intercepts, or a combination of both. 
Introducing a single outlying school also increased the standard error of yoi, the 
effects being more dramatic when the outlier was an extreme slope. 
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How do Extreme Schools Change the Interpretation of Results in School- 
Effectiveness Research?: Effects of Outlying Second-Level Variables in HLM. 

Theoretical Perspective 

Multilevel analysis has recently replaced multiple linear regression, (MLR) 
as the method of choice for school-effects research (Mendro, Webster, Bembry, & 
Orsak, 1995; Webster, Mendro, & Ahnaguer, 1993). Bryk and Raudenbush, 

(1992) give at least two explanations for the appropriateness of multilevel analysis 
for school-effects research. First, random variation and structural effects may 
exist at more than one level, and therefore a correctly specified model is a 
multilevel model in which fixed and random effects can be estimated at each level. 
Second, the assumption of independence of errors in MLR is violated when there 
is intraclass correlation, as one would find in such hierarchically nested data. 

The models for a simple multilevel analysis with one level- 1 and one level- 
2 fixed-effect variable would be; 

Yy = Poj + PljXlj + nj (1) 

for the level- 1 model and 

Poj= Yoo + YoiWj + Uqj (2) 

Pij= Yio + YnWj + uij (3) 

for the level-2 models. The level- 1 model is analogous to a simple regression 
model, where ry is the level- 1 error. The parameters from level- 1 become outcome 
variables in the level-2 models, which are predicted by second-level effects. U(y 
and Ujj are the random error components of level 2. Upon inspection of these 
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models, it is apparent that aberrant or extreme values for Poj and P jj will affect the 
estimation of parameters in the second-level models. These discrepant schools 
could have an undue influence on the estimation of the second-level fixed effects 
of the model. Therefore, Bryk and Raudenbush, (1992) suggested that cross-level 
exploratory analyses be performed when conducting school-effects studies to 
determine if there are schools with outlying intercepts or slopes. 

The effects of outliers have been extensively studied in ordinary least 
squares regression analysis. Rousseeuw and van Zomeren (1990) identify three 
types of outliers: a) vertical outliers in which there is a large residual, b) leverage 
point outliers, which are consistent with the relationships found in the rest of the 
data and c) leverage point outliers, which significantly alter the relationships by 
their inclusion, as well as having high leverage. Outliers exert effects in not only 
the results of the main and interaction-effects significance tests from GLM, but 
also in the estimation of the model parameters (Douzenis & Rakow, 1987; Hecht, 
1991). 

This study is designed to examine the robustness of the estimation of fixed- 
effects in multilevel analysis, when conducting school-effects studies with 
outlying schools. Outlying values for both intercepts and slopes for individual 
schools are modeled separately to determine the effects of extreme values of 
second-level variables on the fixed-effect parameter estimation. 

Data Source 

A total of seven data sets were generated for this study. Data set 1 was 
simulated to serve as the major body of data. This dataset consisted of 250 
schools with 50 students in each school. Three datasets were generated based on 
the original dataset, but containing 10%: a) schools with extreme intercepts, b) 
schools with extreme slopes, or c) schools with extreme intercepts and slopes. 

The extreme intercept is analogous to adding vertical outliers. These were 
generated by increasing the mean approximately 3 standard deviations units 
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higher. The slope outliers were generated by reducing the correlation between X 
and Y from .7 to .4, holding Sx and Sy constant. The dataset size was maintained 
at 250 level 11 observations or schools. The three new datasets were termed 
Outlier-1, OutUer-S, and Outlier-Sl, respectively. Multiple regression diagnostics 
tests of the schools, were examined to determine the most extreme outliers for 
each of the three datasets, as well as to determine that each dataset contained 10% 
outliers. Specifically, the intercept and slope DF Betas were checked to ensure that 
they increased for the outlying observations. 

To produce the last three data sets, extreme outlying observations were 
substituted for observations in the original dataset. One school was removed from 
the original data set, and the most extreme outlying school found in the analysis of 
Outlier-1 was inserted in its place. This new dataset was termed Single-1. This 
process was repeated for OutUer-S, and OutUer-Sl, producing the new datasets 
Single-S, and Single-Sl, respectively. 

Procedures and Results 

A two-level hierarchical linear modeling program (HLM) was used to 
conduct the multilevel analysis for this study. Separate HLM analyses were 
conducted for each dataset. Table 1 presents the descriptive statistics of the seven 
datasets. It can be seen that adding the 10% outlying schools increased the 
standard deviations of the original dataset. However, adding the one outlying 
school had little effect on the standard deviations. 
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Table 1 

Descriptive Statistics 





Outlier 


Variable 


N . 


Mean 


SD 


Minimum 


Maximum 






X 


12,500 


180.04 


9.04 


142.51 


214.02 


No Outlier 




Y 


12,500 


192.03 


13.02 


140.30 


241.76 






W 


250 


5.60 


0.14 


5.21 


5.99 






X 


12,500 


180.04 


9.04 


142.51 


214.02 




Intercept 


Y 


12,500 


197.03 


19.34 


140.30 


282.38 






W 


250 


5.60 


0.14 


5.21 


5.99 






X 


12,500 


180.02 


9.06 


142.51 


214.02 


10% 


Slope 


Y 


12,500 


184.20 


26.89 


73.44 


241.76 


Outliers 




W 


250 


5.60 


0.14 


5.21 


5.99 






X 


12,500 


180.05 


9.07 


142.51 


214.02 




Intercept 


Y 


12,500 


188.27 


17.25 


106.95 


241.76 




& Slope 


W 


250 


5.60 


0.14 


5.21 


5.99 






X 


12,500 


180.04 


9.04 


142.51 


214.02 




Intercept 


Y 


12,500 


192.23 


13.33 


140.30 


276.10 






W 


250 


5.60 


0.14 


5.21 


5.99 






X 


12,500 


180.05 


9.05 


142.51 


214.02 


Single 


Slope 


Y 


12,500 


191.73 


13.87 


94.20 


241.76 


Outlier 




W 


250 


5.60 


0.14 


5.21 


5.99 






X 


12,500 


180.05 


9.04 


142.51 


214.02 




Intercept 


Y 


12,500 


191.89 


13.22 


127.53 


241.76 




& Slope 


W 


250 


5.60 


0.14 


5.21 


5.99 
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Table 2 

Average Regression Coefficient and Reliability Estimation for Level I 





Outlier Type 


Po 


Reliabilitypo 


Pi 


Reliabilitypi 


No. Outlier 




192.00 


0.486 


1.01 


0.043 




Intercept 


197.00 


0.992 


1.01 


0.046 


10% 


Slope 


184.20 


0.997 


0.97 


0.465 


Outliers 


Intercept & Slope 


188.30 


0.986 


0.97 


0.432 




Intercept 


192.20 


0.853 


1.01 


0.053 


Single 


Slope 


191.70 


0.936 


1.01 


0.059 


Outlier 


Intercept & Slope 


191.90 


0.803 


1.01 


0.080 



Table 2 presents the level I regression coefficients and the corresponding 
reliability estimates.^ It can be seen that the reliability estimates for all the 
datasets with outliers were increased. The increase in the school differences in the 
parameter estimates increased iqq which increased the reliability estimates. The 
average parameter estimate for fio changed when outliers were added. This 
occurred for both 10% outliers and single outliers. The average coefficient 
estimate for p, did not change when intercept outliers were added. However, it 
did change the average estimate of p, when 10% slope or slope and intercept 
outliers were added. 
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Table 3 

Regression Coefficient for Level II 





Outlier 

Type 


Yoo 


SE yoo 


Yol 


SE yoi 


y.o 


SE yio 


Yn 


SE yir 


No Outlier 




192.0* 


0.116 


4.51* 


0.804 


1.01* 


0.009 


0.07 


0.066 




Intercept 


197.0* 


0.911 


-10.7 


6.333 


1.01* 


0,010 


0.07 


0.066 


10% 


Slope 


184.2* 


1.498 


1.06 


10.412 


0.97* 


0.013 


0.05 


0.091 


Outliers 


Intercept 
& Slope 


188.3* 


0.726 


2.54 


5.048 


0.97* 


0.013 


0.08 


0.089 , 




Intercept 


192.2* 


0.217 


4.18* 


1.505 


1.01* 


0.010 


0.06 


0.067.:. 


Single 


Slope 


191.7* 


0.328 


2.15 


2.277 


1.01* 


0.010 


0.06 


0.067. 


Outliers 


Intercept 
& Slope 


191.9* 


0,187 


4.21* 


1.301 


1.01* 


0.010 


0.07 


b.0^8 . 



Table 3 shows the regression coefficients estimates for level II, equations 2 
and 3. The standard error estimates of all parameters increased relative to the 

dataset with no outliers. The outliers had little effect on both yn and Yioand did not 
alter the statistical test results. This indicates that the outliers generated under 
these conditions had little effect on the relationship between X and Y when W=0, 

Yio, or the mean difference in the X-Y slopes across levels of W, Yu- However, the 
outliers did have an effect on Yoi, the mean difference in Y across levels of W. 

Both the values for Yoi and its standard error changed such that in all datasets with 
10% outliers the statistical test of Yoi was no longer significant. The net effect of 

the outliers then, was making the tests of Yoi more conservative. For the single 
outlier datasets, only the outlying slope changed the results of the significance 
tests of Yoi. 
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Discussion 

When a cluster of outlying level-II observations are added to a multilevel 
analysis, the effects on the estimation of level II fixed effects can be dramatic. In 
school-effectiveness research, this would be manifested if a cluster of schools had 
extreme means on Y, aberrant relationships between X and Y in a cluster of 
schools, or if a cluster of schools had both extreme means and extreme X-Y 
relationships. Under the conditions investigated in this study the changes in the 
parameter estimates and their standard errors resulted in conservative tests of 
significance. Adding a single outlying school naturally had less of an effect, 
however, results of tests of significance were still changed in the presence of a 
single outlying slope. Further, it is conceivable that similar results could arise for 
single outliers of the intercept and the slope and intercept for more extreme 
outliers. 

Future Research 

This study suggests a more extensive study to ascertain how outliers of 
differing magnitudes affect parameter estimation in multilevel analysis is needed. 
It would also be of interest to study the effects of other types of outliers, such as 
leverage point outhers, on parameter estimation in multilevel analysis. 
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Footnotes 

^Reliability in HLM refers to the ratio of parameter variance to total 
variance. 

Reliability =-^ ) (4) 

'J j=\ 

for each q=0,...,Q (Bryk & Raudenbush, 1992, p.43) and where iqq = parameter 
variance and Vqq = error variance (Bryk & Raudenbush, p.34). 
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